# Model evaluation

ImagenHub is a one-stop repository for standardizing the inference and evaluation of all conditional image generation models. The project first defines seven prominent tasks and creates high-quality evaluation datasets. Second, we build a unified inference pipeline to ensure fair comparisons. Third, we design two human evaluation metrics, semantic consistency and perceptual quality, and establish comprehensive guidelines for evaluating generated images. We train expert reviewers to evaluate model outputs based on the proposed metrics. This human evaluation achieved high inter-rater consistency on 76% of the models. We comprehensively evaluated around 30 models and observed three key findings: (1) The performance of existing models is generally unsatisfactory, with 74% of models scoring lower than 0.5 overall except for text-guided image generation and theme-driven image generation. (2) We examined claims made in published papers and found 83% of the claims to be accurate. (3) Apart from theme-driven image generation, existing automatic evaluation metrics have no Spearman correlation coefficient higher than 0.2. In the future, we will continue to evaluate newly released models and update the rankings to track the progress of the conditional image generation field.

Phixtral-2x2_8 is the first mixed expert model built from two microsoft/phi-2 models, inspired by the mistralai/Mixtral-8x7B-v0.1 architecture. It surpasses the performance of each individual expert model. This model excels in multiple benchmark datasets including AGIEval, GPT4All, TruthfulQA, and Bigbench. It utilizes a customized version of the mergekit library (mixtral branch) with specific configurations. Users can run Phixtral at 4-bit precision on free T4 GPUs via Colab notebooks. The model has 4.46B parameters and utilizes F16 tensor type.